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Abstract 

The  Air  Force  has  installed  a  pilot  program  at  the  Eastern  Range  where  new  IEEE  1588 
enabled  GPS  Timing  receivers  have  been  deployed  at  Cape  Canaveral  Air  Force  Station, 
Kennedy  Space  Center,  and  Patrick  Air  Force  Base  and  are  performance  monitored, 
independently  of  GPS.  This  is  the  first  known  deployment  of  IEEE  1588  where  Master  and 
Slave  IEEE  1588  devices  have  been  installed  on  a  single  Ethernet  network,  over  such  a  large 
area.  The  IEEE  1588  Grandmaster  is  located  in  the  operations  control  center  and  is  disciplined 
by  a  cesium  steered  by  USNO  via  Two-Way  Satellite  Time  Transfer  System.  An  automated 
workstation  at  the  operations  control  center  collects  health  and  status  from  the  GPS  receivers. 
GPS  -  IEEE  1588  difference  data  are  also  collected  to  validate  that  GPS  timing  signals 
provided  to  remote  instrumentation  are  accurate  within  ±  2  microseconds.  This  paper  will 
describe  the  recently  deployed  system  and  share  some  of  the  early  performance  data  collected. 


INTRODUCTION 

The  Eastern  Range  (ER)  has  used  Synchronized  Time  Code  Generators  (STCGs)  to  provide  timing 
signals  to  remote  instrumentation  sites.  For  many  years,  land  line  and  RF  distribution  of  Inter-Range 
Instrumentation  Group  (IRIG)  standard  time-of-year  were  the  synchronizing  signals  of  choice.  In  recent 
years,  many  ER  remote  Timing  sites’  STCGs  have  been  replaced  with  modern  GPS  Timing  Receivers 
capable  of  providing  locally  required  time  codes,  repetition  rates,  and  precision  frequencies.  All  these 
methods  required  the  use  of  portable  clock  to  verify  time  accuracy  of  signals  provided  to  remote 
instrumentation  systems.  Several  efforts  were  made  to  implement  a  scheme  to  verify  the  timing  accuracy 
of  remote  Timing  sites’  clocks  from  a  central  operations  center.  Until  the  introduction  of  the  Institute  of 
Electrical  and  Electronics  Engineers  (IEEE)  1588  Precise  Time  Protocol  (PTP)  standard,  every  scheme 
was  too  expensive  or  inaccurate  to  justify  its  implementation.  This  paper  describes  the  IEEE  1588 
implementation  on  the  ER  intended  to  provide  real-time,  time  accuracy  verification  of  remote  timing 
sites. 
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THE  SYSTEM 

The  Timing  Health  and  Analysis  Monitor  System  (THAMS)  consists  of  the  THAMS  Server/Workstation 
and  the  Grandmaster  IEEE  1588  clock  located  in  the  Morrell  Operations  Center  (MOC),  and 
Member/Slave  IEEE  1588  clocks  located  at  sites  located  at  Cape  Canaveral  Air  Force  Station  (CCAFS), 
Patrick  Air  Force  Base  (PAFB),  and  Kennedy  Space  Center  (KSC).  The  THAMS  Server/Workstation 
and  Grandmaster/Member  IEEE  1588  Clocks  are  connected  via  dedicated  single  fiber-optic  strands, 
Ethernet  switches,  and  media  converters.  The  Grandmaster  IEEE  1588  clock  receives  its  initial 
synchronization  from  the  Global  Positioning  System  (GPS),  and  then  maintains  time  by  accepting  10 
megahertz  (MHz)  from  the  Range  Clock  in  the  MOC,  which  is  synchronized  and  syntonized  to  the 
Department  of  Defense  Master  Clock  (DOD  MC)  at  US  Naval  Observatory,  via  the  Two-Way  Satellite 
Time  Transfer  System.  The  primary  function  of  the  GPS  Receivers  at  remote  instrumentation  sites  is  to 
provide  specification  compliant  time  to  Range  instrumentation.  The  THAMS  enables  operators  to 
remotely  verify  the  accuracy  of  time  provided  to  Range  users  by  comparing  the  times  from  GPS  and 
IEEE  1588  at  each  Site  Clock.  Site  Clock  time  accuracy  measurement  was  expected  to  be  in  the 
microsecond  range. 


THE  COMPONENTS 

The  significant  component  of  the  THAMS  is  the  GPS  Receiver  assembly  that  is  based  on  Symmetricom 
XLi  IEEE  1588  Clock.  This  is  a  commercial-off-the-shelf  item,  which  is  installed  with  an  antenna,  in¬ 
line  amplifier  for  cable  lengths  greater  than  150  feet,  antenna  cable,  two  lightning  arrestors,  and  the 
necessary  hardware  and  adapters  for  installation  and  interface  to  the  existing  building  structure  and  timing 
equipment.  The  GPS  Receiver  assembly  has  a  Grandmaster  IEEE  1588  plug-in  module  in  Bay  4,  a 
Member/Slave  1588  plug-in  module  in  Bay  2,  a  four-channel  sine  wave  frequency  output  plug-in  module 
in  Bay  3,  and  a  GPS  Engine  plug-in  module  in  Bay  1 . 
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Site  Clocks 


Infrastructure 


Range  Clock 


File:  Time  Monitor  Sys  BD  20091 1 1 9A.ppt 


Figure  1 .  TFIAMS  block  diagram. 


THE  SOFTWARE 

The  THAMS  Server/Workstation  provides  the  operator  a  graphical  user  interface  (GUI)  to  monitor, 
configure,  and  observe  status  of  the  GPS  Receivers,  Transition  Networks,  Inc.,  Media  Converters,  Cisco 
IE3000  Switches,  and  Dell  PowerEdge  2950  III  Server.  The  following  software  operating  system  and 
applications  are  installed  on  THAMS  to  provide  for  health,  status,  and  control  of  the  GPS  Receivers  and 
the  Ethernet  components: 

THAMS  Software 

•  Hewlett-Packard  (HP)  NNM  8i 

•  Symmetricom  TimeMonitor  XLi  Software 

•  Symmetricom  TimeMonitor  Analyzer  Software 

•  Rainbow  Technologies  Sentinel 

•  Transition  Networks  Focal  Point  2.1  Management  Software 

•  Hummingbird  Connectivity  Secure  Shell  Management  Console 
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•  Hummingbird  HostExplorer  Open  Session 

•  Dell  OpenManage  Server  Administrator. 

The  Symmetricom  TimeMonitor  XLi  Software  allows  for  monitoring  the  status  of  the  GPS  Receivers  and 
capturing  timing  performance  data.  Once  data  collection  is  initiated,  it  will  continue  without  manual 
intervention  until  an  operator  stops  the  collection,  or  data  storage  limit  is  reached  (approximately  1 1  days 
of  data  is  the  longest  collection  period  to  date).  However,  if  some  event  or  failure  disrupts  collection  and 
prevents  the  normal  closing  of  the  file,  then  all  data  since  the  start  of  collection  are  lost.  The  data 
collected  may  be  saved  to  a  file  for  additional  analysis  with  the  Symmetricom  TimeMonitor  Analyzer 
software  application.  The  collected  data  may  also  be  opened  with  Microsoft  (MS)  Excel  2007  as  a 
delimited  text  file  to  a  text  file  for  further  viewing  and  analysis.  The  Rainbow  Technologies  Sentinel 
application  is  loaded  before  the  hardware  universal  serial  bus  (USB  )  key  is  installed.  The  hardware  USB 
key  is  required  to  open  the  Symmetricom  TimeMonitor  applications. 


THE  SERVER/WORKSTATION 

THAMS  is  implemented  on  a  Dell  PowerEdge  Server  Model  2950  111  running  MS  Windows  2008  Server 
64-Bit  operating  system.  The  computer  has  the  following  components: 

THAMS  Server  Hardware  Components 

•  Rackmount  Chassis  (2 -rack  units  high,  19-inches  wide) 

•  Dual  Core  XEON  Processor:  2.0-gigahertz  (GHz),  1333-megahertz  (MHz)  front-side  bus,  L2 
cache 

•  System  Memory:  8 -gigabyte  single-ranked  Dual  In-line  Memory  Modules 

•  Two  146-  gigabyte  Removable  Hard  Drives 

•  Optical  Drive:  Digital  Versatile  Disc  -  Rewriteable,  Serial  Advanced  Technology 
Attachment,  Internal 

•  Two  Redundant,  Server  Power  Supplies 

•  Two  10/100/1000  Network  Interface  Cards. 

Communication  Equipment© 

•  Cisco  IE3000  Ethernet  Switch  (SW) 

•  Transition  Networks  Media  Converter. 

LOCATION 

Figure  3  shows  a  portion  of  Brevard  County,  Florida.  THAMS  equipment  is  located  at  Kennedy  Space 
Center  (KSC),  Cape  Canaveral  Air  Force  Station  (CCAFS),  and  Patrick  Air  Force  Base  (PAFB).  The 
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figure  shows  the  general  location  of  the  Morrell  Operations  Center  (MOC)  and  the  remote 
instrumentation  sites.  The  system  extends  from  the  Radar  0.134  at  Patrick  Air  Force  base  north  33  miles 
to  the  Playalinda  Beach  Distance  Object  Attitude  Measurement  System  (DOAMS)  north  of  the  National 
Aeronautics  and  Space  Administration  (NASA)  Shuttle  launch  pads.  The  east-west  distance  from  the 
Range  Communications  Bldg  to  the  Radar  19.14  is  over  5  miles.  The  cable  path  from  the  Radar  0.134  to 
the  MOC  is  approximately  15.9  miles.  The  cable  distance  from  the  MOC  to  the  most  distance  site,  the 
Playalinda  Beach  DOAMS,  is  approximately  21.3  miles. 
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Figure  3.  THAMS  general  location  map. 


THE  ENVIRONMENT 

The  environment  is  depicted  outside  the  circle  in  Figure  4.  Two  independent  means  of  time  transfer  from 
IJSNO  provides  assurance  that  systemic  problems  are  detected.  The  GPS  receiver  provides  the  timing 
signals  needed  at  the  remote  instrumentation  site.  The  operator  submits  requests  current  status  and  past 
performance  data.  The  THAMS  responds  by  saving  requested  performance  data  for  later  analysis  and 
display  of  health  and  status  data  for  immediate  assessment. 


347 


42nd  Annual  Precise  Time  and  Time  Interval  (PIT I )  Meeting 


The  block  diagram  shown  in  Figure  5  provides  network  details.  Single  fibers  are  identified  by  the  locally 
assigned  reference  designator.  Full  duplex  media  converter  pairs  convert  the  optical  signals  into  the 
copper  circuits  at  each  Ethernet  Switch  (SW).  Media  converter  and  Ethernet  Switch  management  ports 
have  been  omitted  from  this  figure.  All  remote  site  GPS  receivers  are  two,  three,  or  four  Ethernet 
Switches  from  the  IEEE  1588  Grandmaster.  The  Ethernet  Switches  support  either  IEEE  1588  version  1 
or  version  2.  However,  since  the  GPS  Receivers  can  support  only  IEEE  1588  version  1  at  this  time,  the 
Ethernet  Switches  are  configured  for  IEEE  1588  version  1  support. 
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Figure  4.  THAMS  environment  diagram. 


SYSTEM  ATTRIBUTES 

The  THAMS  provides  ER  Timing  Operators  with  a  central  monitoring  capability  where  remote  timing 
sites  can  be  monitored  for  health,  status,  and  time  accuracy  performance.  The  THAMS  embedded 
Network  Node  Manager  receives  Simple  Network  Management  Protocol  real-time  trap  notification 
messages  from  Ethernet  enabled  devices,  which  are  presented  on  the  THAMS  workstation/server  monitor. 
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With  a  simple  glance  at  the  monitor,  operators  can  determine  the  health  and  status  of  the  network  and  the 
devices  connected  to  it.  Real  time  GPS  -  IEEE  1588  time  difference  data  for  each  remote  site  are  also 
available  on  the  THAMS  monitor.  And  there  are  Web  interfaces  to  the  Ethernet  enabled  devices  that  can 
be  accessed  to  obtain  detailed  health  and  status  and  to  effect  configuration  changes.  This  deployment  of 
IEEE  1588  is  successfully  deployed  over  the  largest  known-to-date  campus  (approximate  size  of  175 
square  miles). 
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Figure  5.  THAMS  block  diagram. 


DATA  COLLECTION 

The  data  are  collected  in  a  .xli  format  compatible  text  format.  See  Figure  6  for  a  GPS  -  IEEE  1588  time 
difference  data  sample.  As  the  figure  shows,  the  time  stamps  are  not  monotonically  increasing.  This  is 
the  result  of  time -interval  values  either  less  than  1  second,  or  greater  than  1  second.  The  first  case  results 
in  two  values  captured  in  the  same  second.  In  the  second  case,  no  value  is  captured  in  the  intervening 
second. 

The  Symmetricom  TimeMonitor  Xli  software  includes  a  utility  TimeMonitor  Analyzer  for  graphing  and 
analysis  of  the  data.  This  tool  was  used  to  graphically  display  the  data  in  the  following  figures. 
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Figure  6.  Data  sample. 


RESULTS  AND  DATA  ANALYSIS 

Figure  7  represents  TimeMonitor  Analyzer  data  collected  from  one  of  the  remote  timing  sites  for  a  period 
of  6  days.  Evidently  there  are  15  time  difference  data  excursions  where  the  data  indicates  that  the  time 
difference  between  GPS  and  TREE  1588  at  the  remote  timing  site  approaches  +  6  milliseconds.  Though 
rare,  these  time  differences  have  been  observed  to  approach  500  milliseconds.  Expanding  the  X  axis  for 
the  data  set,  such  as  Figure  8  for  1  hour/division,  and  Figure  9  for  10  minutes/division,  additional  time 
difference  data  excursions  are  observed.  Not  until  “quiet”  data  are  observed  in  presentation  format  for  1 
minute/division  (Figure  10)  can  data  for  each  time  difference  measurement  be  observed  without 
excursion.  From  these  data,  one  must  conclude  that  the  goal  of  verifying  time  accuracy  performance 
with  less  than  2-microsecond  precision  cannot  be  achieved  in  real  time. 
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Figure  7.  Remote  site  data  (6  days,  12  hours  per  division). 
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Figure  8.  Remote  site  data  (15  hours,  1  hour  per  division). 
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Symmetric  om  TimeMonitor  Analyzer  Phase  Deviation  versus  time  (file*KSCDOAnS-2010_10J2-07_58_22.xU) 
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Figure  9.  Remote  site  data  (1.6  hours,  10  minutes  per  division). 
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SS  Symmetricom  TimeMonitor  Analyzer  Phase  Deviation  versus  time  (file=KSCDOAMS-2010_10_12-07_58_22.xli) 
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Figure  10.  Remote  site  data  (10  minutes,  1  minute  per  division). 


Postprocessing  of  the  time  difference  data  was  performed  using  the  technique  of  removing  outlier  time 
difference  data  that  exceeds  3  times  the  standard  deviation  from  the  mean  value.  This  technique  has  to  be 
applied  for  most  data  sets  2  to  4  times,  effectively  eliminating  1  to  3  percent  of  data  collected,  to 
determine  a  nominal  time  difference  data  set  for  the  remote  timing  sites.  With  the  above  postprocessing, 
remote  timing  site  time  accuracy  performance,  with  a  precision  of  less  than  2  microseconds,  can  be 
achieved  in  most  cases.  However,  time  difference  data  excursions  of  +  8  microseconds  still  are  evident. 
Reference  Figure  1 1 . 
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Figure  1 1 .  Plot  with  outliers  removed  (6  days,  12  hours  per  division). 


POTENTIAL  IMPROVEMENTS 

The  Symmetricom  XLi  GPS  Receiver  lacks  secure  protocols,  which  pose  significant  Information 
Assurance  issues  for  Air  Force  installations.  Symmetricom  reports  that  they  will  ultimately  provide 
secure  protocol  upgrades  for  the  XLi  GPS  Receiver.  However,  at  this  time  the  release  date  for  this 
upgrade  is  unknown. 

The  source(s)  of  the  GPS  -  TFFF  1588  time  difference  data  excursions  are  unknown.  Potentially 
upgrading  the  GPS  Receivers  and  THAMS  network  to  use  IEEE  1588  version  2  will  alleviate  these  data 
anomalies.  Symmetricom  reports  that  IEEE  1588  version  2  for  the  XLi  GPS  Receivers  is  planned. 
However  at  this  time,  the  release  date  for  this  upgrade  is  unknown.  Other  potential  upgrades  should  be 
considered  if  there  is  sufficient  potential  that  the  time  data  excursions  can  be  mitigated. 
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SUMMARY  AND  CONCLUSION 

The  ER  has  deployed  IEEE  1588  over  the  largest  campus  known  by  the  authors  of  this  paper.  Unlike 
many  IEEE  1588  applications  where  IEEE  1588  is  used  to  distribute  precise  time  over  a  network,  the 
main  application  for  this  network  is  to  provide  a  real-time  means  for  monitoring  the  time  accuracy 
performance  of  remote  timing  systems.  These  remote  timing  sites  use  IEEE  1588-enabled  GPS  Receivers 
to  provide  timing  signals  to  ER  instrumentation  across  Cape  Canaveral  Air  Station,  Patrick  Air  Force 
Base,  and  Kennedy  Space  Center.  The  THAMS  network  provides  improved  health,  status,  and  time 
accuracy  performance  monitoring.  In  that,  real-time  monitoring  of  remote  timing  sites  has  been  achieved. 
However,  this  system  has  not  achieved  the  goal  of  monitoring,  in  real  time,  time  accuracy  performance  of 
remote  timing  sites  with  a  precision  of  better  than  2  microseconds.  The  GPS  -  IEEE  1588  time  difference 
data  excursions  are  problematic  and  their  causes  are  unknown  by  the  authors.  Further  investigation  is 


warranted. 

Acronyms  List 

1SLSOB 

1st  Space  Launch  Squadron  Operations  Building  (Squadron  was  disbanded  and 
facility  is  now  called  the  Delta  II  Operations  Building) 

CCAFS 

Cape  Canaveral  Air  Force  Station 

CIF 

Central  Instrumentation  Facility 

DOAMS 

Distant  Object  Attitude  Measurement  System 

DOD  MC 

Department  of  Defense  Master  Clock 

ER 

Eastern  Range 

Gb/s 

Gigabits  per  second 

GPS 

Global  Positioning  System 

GTB 

Guidance  Test  Building 

HP 

Hewlett-Packard 

IEEE 

Institute  of  Electrical  and  Electronics  Engineers 

IP 

Internet  protocol 

IRIG 

Inter-Range  Instrumentation  Group 
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KSC 

Kennedy  Space  Center 

LCC 

Launch  Control  Center 

Mb/s 

megabits  per  second 

MHz 

megahertz 

MC 

Media  Converter 

MOC 

Morrell  Operations  Center 

MS 

Microsoft 

PAFB 

Patrick  Air  Force  Base 

pps 

pulse  per  second 

PTp 

Precise  Time  Protocol 

RF 

radio  frequency 

SLRSC 

Spacelift  Range  System  Contract 

STCG 

Time  Code  Generators 

SW 

Switch  assembly 

Tel-4 

Central  Telemetry  Station 

THAMS 

Timing  Health  and  Analysis  Monitor  System 

TOF 

Test  Operations  Facility 

XY 

Range  Communications  (building) 
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